Joint Calibration of a Multimodal Sensor System for Autonomous Vehicles.

Multimodal sensor systems require precise calibration if they are to be used in the field. Due to the difficulty of obtaining the corresponding features from different modalities, the calibration of such systems is an open problem. We present a systematic approach for calibrating a set of cameras with different modalities (RGB, thermal, polarization, and dual-spectrum near infrared) with regard to a LiDAR sensor using a planar calibration target. Firstly, a method for calibrating a single camera with regard to the LiDAR sensor is proposed. The method is usable with any modality, as long as the calibration pattern is detected. A methodology for establishing a parallax-aware pixel mapping between different camera modalities is then presented. Such a mapping can then be used to transfer annotations, features, and results between highly differing camera modalities to facilitate feature extraction and deep detection and segmentation methods.

Cognitive Relevance Transform for Population Re-Targeting.

This work examines the differences between a human and a machine in object recognition tasks. The machine is useful as much as the output classification labels are correct and match the dataset-provided labels. However, very often a discrepancy occurs because the dataset label is different than the one expected by a human. To correct this, the concept of the target user population is introduced. The paper presents a complete methodology for either adapting the output of a pre-trained, state-of-the-art object classification algorithm to the target population or inferring a proper, user-friendly categorization from the target population. The process is called 'user population re-targeting'. The methodology includes a set of specially designed population tests, which provide crucial data about the categorization that the target population prefers. The transformation between the dataset-bound categorization and the new, population-specific categorization is called the 'Cognitive Relevance Transform'. The results of the experiments on the well-known datasets have shown that the target population preferred such a transformed categorization by a large margin, that the performance of human observers is probably better than previously thought, and that the outcome of re-targeting may be difficult to predict without actual tests on the target population.

Correcting Decalibration of Stereo Cameras in Self-Driving Vehicles.

Camera systems in autonomous vehicles are subject to various sources of anticipated and unanticipated mechanical stress (vibration, rough handling, collisions) in real-world conditions. Even moderate changes in camera geometry due to mechanical stress decalibrate multi-camera systems and corrupt downstream applications like depth perception. We propose an on-the-fly stereo recalibration method applicable in real-world autonomous vehicles. The method is comprised of two parts. First, in optimization step, external camera parameters are optimized with the goal to maximise the amount of recovered depth pixels. In the second step, external sensor is used to adjust the scaling of the optimized camera model. The method is lightweight and fast enough to run in parallel with stereo estimation, thus allowing an on-the-fly recalibration. Our extensive experimental analysis shows that our method achieves stereo reconstruction better or on par with manual calibration. If our method is used on a sequence of images, the quality of calibration can be improved even further.

Using a Situation Awareness Approach to Identify Differences in the Performance Profiles of the World's Top Two Squash Players and Their Opponents.

PURPOSE: The pressure exerted on a squash player is a consequence of the quality of a shot coupled with the ability of the player to return the ball, namely, the coupling of the two players' situation awareness (SA) abilities. SA refers to an awareness of all relevant sources of information, the ability to synthesize this information using domain knowledge and the ability to physically respond to a situation. METHODS: Matches involving the two best players in the world (n = 9) at the 2011 Rowe British Grand Prix, held in Manchester, United Kingdom were recorded and processed using Tracker software. Shot type, ball location, players' positions on court and movement parameters between the time an opponent played a shot prior to the player's shot to the time of the opponent's following shot were captured 25 times per second. All shots (excluding serves and rally ending shots) produced five main SA clusters, similar to those presented by Murray et al. (2018), except a greater proportion of shots were categorized in the greater pressure clusters and less in the lower pressure ones. RESULTS: Individual matches were presented using cluster performance profile infographics which demonstrated how individual player's performance profiles differed between matches. CONCLUSION: It is suggested that it is the coupling, of the two player's behaviors, that makes the examination of tactics so challenging. This inherently means that performance profiles vary in subtle ways, making consistent profiles that are independent of the opponent very unlikely for elite players. This approach should be further modified to determine within match changes in performance.

Quantitative Contact-Less Estimation of Energy Expenditure from Video and 3D Imagery.

Measurement of energy expenditure is an important tool in sport science and medicine, especially when trying to estimate the extent and intensity of physical activity. However, most approaches still rely on sensors or markers, placed directly on the body. In this paper, we present a novel approach using a fully contact-less, fully automatic method, that relies on computer vision algorithms and widely available and inexpensive imaging sensors. We rely on the estimation of the optical and scene flow to calculate Histograms of Oriented Optical Flow (HOOF) descriptors, which we subsequently augment with the Histograms of Absolute Flow Amplitude (HAFA). Descriptors are fed into regression model, which allows us to estimate energy consumption, and to a lesser extent, the heart rate. Our method has been tested both in lab environment and in realistic conditions of a sport match. Results confirm that these energy expenditures could be derived from purely contact-less observations. The proposed method can be used with different modalities, including near infrared imagery, which extends its future potential.

Using a situation awareness approach to determine decision-making behaviour in squash.

Situation awareness (SA) refers to the awareness of all relevant sources of information, an ability to synthesise this information using domain knowledge gained from past experiences and the ability to physically respond to a situation. Expert-novice differences have been widely reported in decision-making in complex situations although determining the small differences in expert behaviour are more elusive. This study considered how expert squash players use SA to decide on what shot to play. Matches at the 2010 (n = 14) and 2011 (n = 27) Rowe British Grand Prix were recorded and processed using Tracker software. Shot type, ball location, players' positions on court and movement parameters between the time an opponent played a shot prior to the player's shot to the time of the opponent's following shot were captured 25 times per second. Six SA clusters were named to relate to the outcome of a shot ranging from a defensive shot played under pressure to create time to an attempted winner played under no pressure with the opponent out of position. This new methodology found fine-grained SA differences in expert behaviour, even for the same shot type played from the same court area, beyond the usual expert-novice differences.

Effects of rule changes on physical demands and shot characteristics of elite-standard men's squash and implications for training.

The physical demands and rally characteristics of elite-standard men's squash have not been well documented since recent rule changes (scoring and tin height). This information is needed to design optimal training drills for physical conditioning provided here based on an analysis of movement and shot information. Matches at the 2010 (n = 14) and 2011 (n = 27) Rowe British Grand Prix were analysed. Rallies were split into four ball-in-play duration categories using the 25th (short), 75th (medium), 95th percentiles (long) and maximum values. Cohen's d and chi-squared tests of independence evaluated effects of rally and rule changes on patterns of play. The proportion of long, middle and short shots was related to the duration of the rally with more shots played in the middle and front of the court in short rallies (phi = 0.12). The frequencies of shots played from different areas of the court have not changed after the adoption of new rules but there is less time available to return shots that reflect the attacking nature of match play for elite-standard men players. Aspiring and current elite-standard players need to condition themselves to improve their ability to cope with these demands using the ghosting patterns presented that mimic demands of modern match play.

Fast Image-Based Obstacle Detection From Unmanned Surface Vehicles.

Obstacle detection plays an important role in unmanned surface vehicles (USVs). The USVs operate in a highly diverse environments in which an obstacle may be a floating piece of wood, a scuba diver, a pier, or a part of a shoreline, which presents a significant challenge to continuous detection from images taken on board. This paper addresses the problem of online detection by constrained, unsupervised segmentation. To this end, a new graphical model is proposed that affords a fast and continuous obstacle image-map estimation from a single video stream captured on board a USV. The model accounts for the semantic structure of marine environment as observed from USV by imposing weak structural constraints. A Markov random field framework is adopted and a highly efficient algorithm for simultaneous optimization of model parameters and segmentation mask estimation is derived. Our approach does not require computationally intensive extraction of texture features and comfortably runs in real time. The algorithm is tested on a new, challenging, dataset for segmentation, and obstacle detection in marine environments, which is the largest annotated dataset of its kind. Results on this dataset show that our model outperforms the related approaches, while requiring a fraction of computational effort.

A new method for assessing squash tactics using 15 court areas for ball locations.

Tactics in squash have typically been assessed using the frequency of different shot types played at different locations on the court either without reference to other relevant information or on the basis of the preceding shot. This paper presents a new squash specific method for categorizing court locations in which the ball was played, a novel techniques for assessing the reliability of this method and presents typical shots responses in these new areas controlled for preceding shot as well as the time between shots and the handedness of the players. Twelve games were viewed using the SAGIT/Squash software and 2907 shots viewed a second time from a video image taken from behind the court with an overall agreement of 88.90% for the court location data and 99.52% for shot type. 3192 shots from 9 matches from the 2003 World Team Championships were analyzed in SAGIT/Squash. In the court areas analyzed between 2 and 7 shot responses were predominant suggesting tactical patterns were evident. This was supported by differences evident between shot responses played from the two back corners where the backhand side was characterized by a predominance of straight drives whereas straight and crosscourt drives were played on the forehand side. These results tended to confirm that tactics i.e., consistent shot types, are played although these are only apparent when factors that determine shot selection are accounted for. This paper has controlled for some of these factors but others need to be considered e.g., if individual player profiles are to be ascertained.

A time-motion analysis of turns performed by highly ranked viennese waltz dancers.

Twenty-four dance couples performing at the 2011 IDSF (International DanceSport Federation) International Slovenia Open were divided into two groups: the first twelve placed couples (top ranked) and the last twelve placed couples (lower ranked). Video recordings were processed automatically using computer vision tracking algorithms under operator supervision to calculate movement parameters. Time and speed of movement were analysed during single natural (right) and reverse (left) turns performed during the Viennese waltz. Both top and lower ranked dancers tended to perform similar proportionate frequencies of reverse (≈ 35%) and natural (≈ 65%) turns. Analysis of reverse turns showed that the top ranked dancers performed less turns on a curved trajectory (16%) than the lower ranked dancers (33%). The top ranked couples performed all turns at similar speeds (F = 1.31, df = 3, p = 0.27; mean = 2.09m/s) all of which were significantly quicker than the lower ranked couples (mean = 1.94m/s), the greatest differences found for reverse turns (12.43% faster for curved trajectories, 8.42% for straight trajectories). This suggests that the ability to maintain a high speed in the more difficult turns, particularly the reverse turns on a curved trajectory, results in the overall dance appearing more fluent as the speed of movement does not fluctuate as much. This aspect of performance needs to be improved by lower ranked dancers if they wish to improve rating of their performance. Future research should determine which factors relate to the speed of turns.

The effect of court location and available time on the tactical shot selection of elite squash players.

No previous research in squash has considered the time between shots or the proximity of the ball to a wall, which are two important variables that influence shot outcomes. The aim of this paper was to analyse shot types to determine the extent to which they are played in different court areas and a more detailed analysis to determine whether the time available had an influence on the shot selected. Ten elite matches, contested by fifteen of the world's top right handed squash players (age 27 ± 3.2, height 1.81 ± 0.06 m, weight 76.3 ± 3.7 kg), at the men's World Team Championships were processed using the SAGIT/Squash tracking system with shot information manually added to the system. Results suggested that shot responses were dependent upon court location and the time between shots. When these factors were considered repeatable performance existed to the extent that one of two shots was typically played when there was limited time to play the shot (< 1.20s). For example, it was clear that when players did not have a lot of time to hit the ball (low time i.e. < 1.06s, and mid time i.e. 1.06 - 1.20s) in the front left corner close to the side wall, the crosscourt lob was used frequently (44.30% and 36.31% respectively) whereas when there was more time this shot was seldom used (13.64%). Consequently variant and invariant behaviour were shown to exist in elite squash although for the first time it was suggested that the availability of time to play a shot contributed to which of these behaviours was evident. This analysis could be extended by adopting a case study approach to see how individual differences in strategy and tactics affect shot selections. Key pointsPrevious research has suggested that a playing strategy, elements decided in advance of the match, may be evident for elite players by examining court location and preceding shot type, however these parameters alone are unlikely to be sufficient predictors.At present there is no known analysis in squash, or indeed in any of the racket sports, that has quantified the time available to respond to different shot types. An understanding of the time interval between shots and the movement characteristics of the player responding to different shots according to the court positions might facilitate a better understanding of the dynamics that determine shot selection.Some elements of a general playing strategy were evident e.g. predominately hitting to the back left of the court, but tactical differences in shot selection were also evident on the basis of court location and time available to play a shot.

Movement characteristics of elite tennis players on hard courts with respect to the direction of ground strokes.

Previous studies of movement characteristics in tennis have considered the effect of playing surface but have assumed that playing strategies are simply determined by the surface as opposed to being under an individual's control. This study considered the selection of cross court or down the line ground strokes as being indicative of playing strategy and measured the outcome of playing these shots in terms of the opponent's movements. Matches (N = 8) at the 2011 ATP tournament 500 Valencia were recorded and analysed using SAGIT, a computer vision tracking system that allowed both players' movements to be tracked automatically, albeit with operator supervision. The data was split into (N = 188) games for analysis purposes and these lasted a median 174.24 seconds with active time (ball in play) a median proportion of 34.89% (IQR = 10.64%) of total time. During the active time losers of games tended to cover less distance (median = 80.17 m), move quicker (median = 1.38 m·s(-1)), spend more time in the defensive zones (median = 14.24 s) and less in the offensive zones (median = 44.74 s). These results suggested that game winners tended to dominate game losers, forcing them to exhibit behaviors typically associated with a defensive strategy. Defensive and offensive strategy are not well defined currently and future investigations should consider movements in relation to individual shots, in particular their velocities, at the rally level and by different individuals to better understand successful performance. Key PointsDuring the active time losers of games covered less distance, moved quicker, spend more time in the defensive zones and less in the offensive zones.These results suggested that game winners tended to dominate game losers, forcing them to exhibit behaviors typically associated with a defensive strategy.There are no differences between the proportion of cross court shots and down the line shots played by game winners and game losers.Future research should consider individual shots at the rally level to better understand successful performance and ultimately strategy.

Tracking by identification using computer vision and radio.

We present a novel system for detection, localization and tracking of multiple people, which fuses a multi-view computer vision approach with a radio-based localization system. The proposed fusion combines the best of both worlds, excellent computer-vision-based localization, and strong identity information provided by the radio system, and is therefore able to perform tracking by identification, which makes it impervious to propagated identity switches. We present comprehensive methodology for evaluation of systems that perform person localization in world coordinate system and use it to evaluate the proposed system as well as its components. Experimental results on a challenging indoor dataset, which involves multiple people walking around a realistically cluttered room, confirm that proposed fusion of both systems significantly outperforms its individual components. Compared to the radio-based system, it achieves better localization results, while at the same time it successfully prevents propagation of identity switches that occur in pure computer-vision-based tracking.

A two-stage dynamic model for visual tracking.

We propose a new dynamic model which can be used within blob trackers to track the target's center of gravity. A strong point of the model is that it is designed to track a variety of motions which are usually encountered in applications such as pedestrian tracking, hand tracking, and sports. We call the dynamic model a two-stage dynamic model due to its particular structure, which is a composition of two models: a liberal model and a conservative model. The liberal model allows larger perturbations in the target's dynamics and is able to account for motions in between the random-walk dynamics and the nearly constant-velocity dynamics. On the other hand, the conservative model assumes smaller perturbations and is used to further constrain the liberal model to the target's current dynamics. We implement the two-stage dynamic model in a two-stage probabilistic tracker based on the particle filter and apply it to two separate examples of blob tracking: 1) tracking entire persons and 2) tracking of a person's hands. Experiments show that, in comparison to the widely used models, the proposed two-stage dynamic model allows tracking with smaller number of particles in the particle filter (e.g., 25 particles), while achieving smaller errors in the state estimation and a smaller failure rate. The results suggest that the improved performance comes from the model's ability to actively adapt to the target's motion during tracking.

A multistage registration method using texture features.

We present a novel, multistage registration method based on Laws' texture features. In general, a large number of texture features may be extracted from the original intensity images. For each of the texture features, a criterion function that measures the similarity between the images may be derived. The proposed registration method consists of two major steps. In the first step, a dataset of images with the corresponding gold standard is used. In this step, the selection and ranking of the texture features for registration is made. The selection and ranking of the features is based on their robustness, accuracy, and capture range. The selected features are then entered in the second step, where the actual registration is performed using a sequence of registration stages. Our method is based on the selection of the most robust feature for the first registration stage and the selection of accurate feature(s) for the subsequent stages. The texture features are daisy-chained so that the accuracy of the previous feature is sufficient for the capture range of the next feature. We tested our method on 11 2D image pairs containing digital reconstructed radiographs and electron portal imaging modalities, which were difficult to register using intensity features alone. With our method, we have successfully registered 75% of the initial displacements, ranging from 5 to 7.5 mm, with the target-registration error below 3 mm, whereas the traditional intensity-based approach delivered only 15% successfully registered cases.

Tactical use of the T area in squash by players of differing standard.

The importance of dominating the T in squash is recognized by coaches and players but there has been little formal investigation of this aspect of tactical play. Consequently, the aim of this research was to analyse player occupancy of a T area, to establish whether there are differences between winners and losers of games at different playing standards. An automated player-tracking system, with operator supervision and intervention, captured players' movements during matches at the World Team Championships (n = 11), the Slovenian National Championships (n = 11), and a local tournament (n = 15). Frequency of occupying the T area at the moment opponents played their shot best discriminated playing standard. Winners spent a greater proportion of total playing time in the T area than losers (P < 0.001), except during closely contested games. The results suggest that time in the T area indicates dominance of rallies. Future studies need to consider both between-group (playing standard) and within-game (individual player standard) differences, as both were shown to influence the time players spent in the T area.

Observation and analysis of large-scale human motion.

Many team sports include complex human movement, which can be observed at different levels of detail. Some aspects of the athlete's motion can be studied in detail using commercially available high-speed, high-accuracy biomechanical measurement systems. However, due to their limitations, these devices are not appropriate for studying large-scale motion during a game (for example, the motion of a player running across the entire playing field). We describe an alternative approach to studying such large-scale motion, and present a video-based, computer-aided system, developed specifically for the purpose of acquiring large-scale motion data. The baseline of our approach consists of sacrificing much of the spatial accuracy and temporal resolution of widely used biomechanical measurement systems, to obtain data on human movement that span large areas and long intervals of time. Data can be obtained for each of the observed athletes with reasonable amount of operator work. The system was developed using the recordings of a handball match. Several field tests were performed to assess measurement error, including comparison to one of the widely available biomechanical measurement systems. With the help of the system presented, we could obtain position data for all 14 handball players on a 40 x 20 m large court with RMS error better than 0.6 m, covering 1 h of action. Several results, obtained during the handball match study are presented, in order to highlight the importance of large-scale motion acquisition.